ALLOCATION  OF  RESEARCH  LABORATORY  RESOURCES: 
A  STUDY  OF  RESOURCES  ALLOCATION  USING  CRITERIA 
OF  MILITARY  SIGNIFICANCE  AND  INVESTMENT  RISK 


Warren  James  Millard 


Library 

Naval  Postgraduate  School 

Monterey,  California  93940 


iUl\ii 


QUIi 


1 


1 1 


4fcsBss»'    il 


\2 


r—— "» 

1 

ALLOCATION 

OF  RESEARCH  LABORATORY 

RESOURCES : 

i    A 

STUDY  OF  RESOURCES  ALLOCATION  USING  CRITERIA  OF 

MILITARY 

SIGNIFICANCE  AND  INVESTMENT  RISK 

by 

Warren  James  Millard 

Thes- 

Ls  Advisor: 

J.  K.  Ariraa 

March  1973 


7153/2$ 


kpptwvtd  ^on.  public  h.<LL<uoJ><i;   dUtnxbuti.on  UYiLunltzd. 


Allocation  of  Research  Laboratory  Resources: 
A  Study  of  Resources  Allocation  using  Criteria  of 
Military  Significance  and  Investment  Risk 


by 


Warren  James  Millard 
Lieutenant,  United  States  Navy 
B.S.,  United  States  Naval  Academy,  1967 


Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN  OPERATIONS  RESEARCH 


from  the 

NAVAL  POSTGRADUATE  SCHOOL 
March  1973 


//< 


c./ 


Library 

Naval  Postgraduate  School 

Monterey,  California  93940 


ABSTRACT 

The  problem  is  defined  as  examining  a  list  of  the 
research  projects  presently  sponsored  by  the  Department 
of  the  Navy  under  GOR-43  and  analyze  the  ordered  project 
listings  as  ranked  by  panels  composed  of  persons  with 
interest  in  those  projects. 

Data  consisted  of  project  priority  rankings  of  193 
judges  composing  9  separate  panels,  each  judge  ranking 
the  projects  twice  in  priority  order;  once  considering 
the  practical  significance  of  the  project  and  once  con- 
sidering the  possibility  of  a  successful  research  effort 
Project  rankings  of  judges  within  each  panel  were  scaled 
using  the  FORD  procedure  and  the  9  panel  rankings  then 
investigated  for  similarities  using  methods  of  cluster 

analys  is . 

The  results  showed  that  no  2  judge  panels  submitted 
highly  correlated  rankings  of  the  18  projects,  but  that 
under  each  criterion  several  project  clusters  were  evi- 
dent, indicating  equal  research  emphasis  desired  by  the 
panels . 
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I.   INTRODUCTION 


There  exists  within  the  Department  of  the  Navy,  many 
organizations  with  varying  convictions  about  which  Navy 
Personnel  Research  and  Navy  Training  Research  Programs 
should  receive  priority  when  allocating  research  funds 
and  research  time.   The  diverse  opinions  of  program  empha- 
sis are  due  partly  to  the  different  compositions  of  the 
groups,  partly  to  the  different  functions  of  the  groups 
and  partly  to  the  relationships  of  the  groups  to  other 
organizations  both  inside  and  outside  the  Department  of 
the  Navy. 

Undoubtedly,  those  who  make  the  final  decisions  have 
the  greater  number  of  relevant  facts  upon  which  to  base 
their  decision.   They  probably  exert  greater  effort  to 
the  decision-making  process  itself  than  their  critics 
might.   Thus  their  decisions  should  be  the  best  "all  fac- 
tors considered",  decisions.   Nevertheless,  an  examination 
of  the  diversity  of  opinions  among  different  groups 
should  help  their  decision-making  process  by  providing 
additional  information  and  by  pointing  out  areas  where 
closer  agreement  among  various  groups  would  promote  the 
research  effort. 

The  objective  of  this  paper  is  to  draw,  from  various 
groups,  data  concerning  which  programs  they  feel  should 
be  studied,  analyze  the  group  decisions  and  examine 


similarities  and.  dissimilarities  of  their  priority  pro- 
ject choices . 

The  nature  of  the  problem  is  two-fold. 

1.  Compare  the  groups  by  comparing  their  choices  of 
which  projects  should  receive  priority  when  allocating 
research  laboratory  resources.   Establish  communities  of 
judge  groups  whose  priority  rankings  displayed  a  degree 
of  similarity. 

2.  Examine  a  list  of  projects  currently  being 
studied  by  the  Bureau  of  Naval  Personnel  and  associated 
research  laboratories.   Determine  if  there  exists  projects 
which  inter-group  consensus  shows  should  receive  either 
high  or  low  priority  when  allocating  research  laboratory 
resources . 

The  method  chosen  to  compare  the  judge  groups  was  by 
means  of  a  two-dimensional  plot  where  inter-point  dis- 
tances on  the  plot  are  a  monotonic  function  of  the  judge 
group  similarities.   More  specifically, 

r  .  .  >  r  .,  =>   d  .  .  <  d  .. 
ij     jk       ij     jk 

r..  =  a  correlation  measure  between  groups  i  and  j. 

d. .  =  the  inter-point  distance  for  points  i  and  j  on  the 

two-dimensional  plot. 

By  examining  the  resulting  plots  using  methods  of 
cluster  analysis,  those  judge  groups  whose  project  prior- 
ity rankings  compared  favorably  were  readily  evident. 


The  methods  for  comparing  the  various  projects  were 
the  same  as  for  the  judge  groups  except  that  a  dissimi- 
larity vice  correlation  measure  was  utilized  for  inter- 
object  comparison.   The  technique  for  determining  the 
plot  accepted  either  statistic  as  input.   The  resulting 
plot  displayed  the  projects  as  an  inverse  monotonic  func- 
tion of  the  dissimilarity  measure.   Specifically, 
t  .  .  <  t  ..  =>   d.  .  <  d  ., 

t.  .  =  the  measure  of  dissimilarity  between  projects  i  and 
j.   d..  =  the  inter-point  distance  for  points  i  and  j  on 
the  two-dimensional  plot. 

Methods  of  cluster  analysis  discerned  those  projects 
which  received  similar  rankings  on  the  priority  scales 
of  the  judge  groups. 

Comparison  of  the  mean  ranking  for  each  project  over 
all  judge  groups  showed  which  projects  were  ranked  high, 
low,  etc.   The  variance  of  the  mean  ranking  for  each  pro- 
ject indicated  the  consistency  of  its  ranking  by  the 
judge  groups,  e.g.,  the  projects  with  the  lowest  rank 
variance  were  ranked  most  consistently  by  all  groups. 


II.   DATA  DESCRIPTION 

The  data  source  for  this  study  consisted  of  approxi- 
mately two-hundred  individual  judges,  each  ranking  the 
same  eighteen  projects  on  a  priority  ladder.   That  pro- 
ject which  the  judge  felt  should  receive  top  priority  was 
ranked  first;  the  second  most  important  was  ranked  second, 
etc.   In  addition,  each  judge  was  allowed  to  indicate 
programs  which  he  believed  should  receive  equal  emphasis 
by  placing  them  (up  to  six)  in  the  same  group.   To  in- 
crease the  validity  of  the  data,  judges  were  asked  to 
rank  only  the  projects  which  they  felt  knowledgeable 
about  and  competent  to  rank. 

Each  judge  produced  two  priority  ladders,  one  ranked 
using  the  criterion  of  Military  Significance  of  the  pro- 
ject and  one  ranked  using  the  criterion  of  Investment 
Risk  of  the  project.   The  two  criteria  are  explained  in 
the  following  paragraph. 

A.   PROJECT  RANKING  CRITERIA 

The  two  major  considerations  when  deciding  upon  which 
projects  to  expend  research  resources  are  the  potential 
Military  Significance  of  the  program  and  the  Investment 
Risk  of  the  program.   The  judging  groups  of  this  study 
were  asked  to  construct  priority  ladders  of  the  con- 
sidered programs  under  each  of  those  two  considerations. 


1 .  Criterion  A.   Military  Significance 

Military  Significance  refers  to  the  needs  of  the 
Navy.   The  projects  were  ranked  on  the  basis  of  v/hich 
would  yield  the  greatest  payoff  to  the  Navy  if  the  re- 
search was  successful.   Those  projects  thought  to  have 
high  military  significance  were  ranked  high  and  so  on, 
down  to  those  of  low  military  significance.   The  proba- 
bility of  a  solution  being  achieved  was  not  considered 
under  this  criterion,  only  the  significance  of  the  solu- 
tion if  it  were  achieved. 

2 .  Criterion  B.   Investment  Risk 

Investment  Risk  refers  to  the  probability  of  a 
successful  research  effort  regardless  of  the  value  to 
the  Navy  of  the  payoff.   The  project  orderings  repre- 
sented each  judge's  belief  of  which  problems  had  a  high 
probability  of  being  successfully  solved  (ranked  high) 
and  which  had  a  lower  probability  of  being  solved  (ranked 
lower).   A  research  effort  was  defined  as  successful  if 
a  solution  to  problems  or  significant  improvements  in 
the  present  state  could  be  developed  within  a  period  of 
ten  years  . 

B.   THE  JUDGE  GROUPS 

The  judge  groups  chosen  for  this  study  represent 
those  organizations  which  are  vitally  interested  in  the 
allocation  of  research  resources  to  the  various  projects. 
The  list  of  the  nine  groups  is  contained  in  Appendix  A. 
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They  represent  a  cross-section  of  persons  directly  con- 
nected with  the  Bureau  of  Naval  Personnel  and  with  the 
Naval  Personnel  Research  Laboratories.   Both  managers  and 
professional  workers  are  included  as  judges.   In  addition, 
group  number  7  is  composed  of  persons  who  are  users  of 
research  products;  group  number  8,  evaluators  of  research 
efforts;  and  group  9,  contracted  workers  on  research  pro- 
grams.  Accordingly,  none  of  these  last  three  groups  is 
directly  involved  in  selecting  projects  for  emphasis  in 
research . 

C.   THE  PROJECTS  STUDIED 

The  projects  selected  for  study  are  ones  of  current 
high  interest  within  the  Department  of  the  Navy.   Appen- 
dix B  lists  and  describes  the  projects.   One  of  the 
objectives  of  this  study  was  to  determine  if  the  judge 
groups  felt  that  progr am  priority  lists  should  be  re- 
assessed.  The  projects  considered  represent  all  those 
that  Personnel  Logistics  (General  Operating  Requirement 
43)  is  currently  concerned  with,  plus  the  additional 
project  of  R  &  D  Resources  Management  (No.  16). 
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III.   CALCULATIONS  ON  DATA 

The  raw  data  submitted  by  each  individual  judge  was 
in  the  form  of  two  priority  ladders,  one  for  each  rank- 
ing criterion.   To  facilitate  comparing  the  judge  groups, 
a  single  composite  ranking  of  projects  for  each  group, 
which  represented  the  preferences  of  each  judge  in  that 
group,  had  to  be  derived. 

A.   THE  FORD  PROCEDURE 

The  technique  for  developing  a  composite  ranking  for 
each  judge  group  is  known  as  the  FORD  PROCEDURE  [Ref.  1]. 
Basically,  the  program  determines  a  weighted  value  for 
each  project.   The  weight  statistic  is  calculated  from 
a  win-loss  matrix  by  solving  iteratively  the  equation 


n+1 

a  .  . 

Vi 

a  .  .    +    a  .  . 
ij             .11 

n       ,       n 

w  .       +   w  . 

i               J 

where 

a.  .  =  the  number  of  times  project  i  was  preferred  to 
ij  r       j  r 

project  j  by  all  judges  in  the  group. 

w.  =  the  weighted  value  assigned  to  project  i  on  the  n  t_h 
iteration . 

w.  =  the  win  percentage  for  project  i  on  the  first  itera- 
tion . 
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It  should  be  noted  that  the  FORD  PROCEDURE  produces 
a  composite  judgment  that  reflects  the  contribution  of 
individual  judges  according  to  the  number  of  judgments 
he  makes . 

Several  commendable  characteristics  which  illustrate 
why  the  FORD  PROCEDURE  is  especially  applicable  to  this 
study  are, 

1.  A  judge  or  ranker  needed  not  rank  all  the  pro- 
jects,  only  those  he  felt  knowledgeable  about  and  compe- 
tent to  rank. 

2.  Each  judge  could  make  his  ranking  as  coarse  or 
as  fine  as  he  desires.   There  was  no  restriction  on  the 
number  of  judgmental  categories  he  may  use. 

3.  A  judge  had  no  requirement  for  a  fixed  distribu- 
tion of  projects  among  the  categories  except  that  no  more 
than  one-third  of  the  projects  could  be  placed  in  any  one 
category . 

4.  A  judge  could  indicate  projects  of  tied  ranking 
by  placing  them  in  the  same  category. 

The  input  to  the  FORD  PROCEDURE,  as  run  on  the  IBM/ 
360  computer,  was  composed  of  each  individual  judge's 
ranking  of  some  or  all  of  the  eighteen  projects,  ordered 
under  Criterion  A  or  B.   The  program  considered  all  judge 
rankings  of  judges  in  one  group  and  calculated  a  weighted 
value  for  each  project.   By  ordering  the  projects  with 
the  highest  weight  first,  a  single  composite  ranking  of 
projects  representing  all  the  judges  in  that  group  was 
obtained . 

13 


B.   DATA  MATRIX  TRANSFORMATION 

After  computing  a  composite  ranking  for  each  group, 
the  data  for  analysis  had  been  reduced  to  an  18  x  18 
matrix  M.    'Rank  positions'  were  labels  on  the  ordinate 
and  'judge  groups'  were  the  labels  on  the  abscissa.   One 
additional  ranking  was  added  to  the  data  matrix  as  column 
10.   It  represented  the  projects  ranked  high  to  low  as 
reflected  by  current  fiscal  year  funding.   Columns  1 
through  9  of  matrix  M  represented  the  priority  rankings 
of  the  projects  by  the  judge  groups  under  the  A  criterion. 
Columns  11  through  19  represented  the  priority  rankings  of 
the  judge  groups  1  through  9  respectively,  under  criterion 
B.   Element  m. .  showed  which  project  was  ranked  in  the 
i  _th  position  by  judge  group  j.   Appendix  C  contains 
matrix  M. 

To  enable  easier  calculation  of  correlation  coeffi- 
cients between  judge  groups  and  between  projects,  the 
matrix  M  was  transformed  to  a  matrix  N,  18  x  19.   Matrix 
M  had  'project  number'  on  the  ordinate  and  'judge  group' 
on  the  absicssa.   Thus  the  element  n. .  represented  the 
ranking  of  project  i  by  judge  group  j. 
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C.   CORRELATION  BETWEEN  JUDGE  GROUPS 

From  ufatrix  N  the  correlation  between  judge  groups 
was  calculated  using  the  formula 


r  .  . 


,     6Z  (d..) 
1  "    u  1J 


N(N  -1) 


where 

d.  .  =  the  difference  between  the  ranking  of  project  k  by 
judge  groups  i  and  j. 
N  =  the  number  of  ranks. 

k  =  an  index  to  facilitate  stepping  through  each  of  the 
18  projects.   The  correlation  coefficient  r. .  has  the  ad- 
vantage  of  not  requiring  any  assumption  on  the  distribu- 
tion of  the  data.   Matrix  R  in  Appendix  D  displays  the 
r  .  .  values . 

D.   PROJECT  COLLATION 

A  statistic  was  required  that  would  quantify  the  rela- 
tionship of  project  i  to  project  j.   It  was  decided  that 
the  best  method  was  to  compare  their  respective  rankings 
by  the  judge  groups.   By  calculating  a  difference  in  the 
mean  ranking  of  each  project,  no  consideration  would  be 
given  to  the  variance  of  the  rankings.   Note  that  if  pro- 
jects r  and  s  with  mean  rank  difference  equal  to  X  have 
small  rank  variance,  they  are  'farther  apart'  than  projects 
u  and  v  with  a  difference  in  mean  ranks  equal  to  X  and  with 


15 


large  rank  variance.   Therefore,  the  statistic  t  .  was 
developed  and  used  as  a  dissimilarity  measure. 


t.  .  = 


MR. 


MR 


^      /(STD. ) (STD) 


where 

MR.  =  the  mean  rank  of  project  i  over  the  judge  group 
rankings . 

STD.  =  the  standard  deviation  of  the  rankings  of  project 
i.  Project  mean  ranks,  standard  deviation  of  the  rank- 
ings, and  the  tabled  t.  .  statistic  are  shown  in  Appendix 
E.  The  smallest  value  for  standard  deviation  of  project 
ranking  was  1.68;  therefore,  the  problem  of  a  very  small 
STD.  value  disproportionately  affecting  the  t..  statis- 
tic  was  not  encountered. 
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IV .   MULTIDIMENSIONAL  SCALING  ANALYSIS 

The  method  chosen  to  compare  the  various  judge  groups 
and  to  compare  the  projects  was  multidimensional  scaling 
analysis.   The  particular  technique  employed  was  non- 
metric  Smallest  Space  Analysis  (SSA),  and  has  evolved 
through  studies  by  Shepard,  Kruskal,  Guttman  and  Lingoes 
[Refs.  2-6],   The  computer  program  used  was  developed  at 
the  University  of  Michigan  by  Guttman  and  Lingoes  and  is 
designated  Smallest  Space  Analysis  -  I  (SSA-I)  [Ref.  7]. 
The  program  finds  a  set  of  coordinates  for  each  of  the  n 
points  (n  judge  groups)  in  an  m  dimensional  Euclidean 
space  such  that  the  distance  between  points  is  a  mono- 
tonic  function  of  inter-judge  group  correlation.   More 
precisely,  given  a  set  of  n  objects  for  which  there  is  a 
defined  distance  function,  such  as  object  correlation 
ranks  p..  on  the  n(n-l)/2  pairs  of  points,  find  that  set 
of  real  numbers,  x.   (i  =  1,  2,..,  n;  a  =  1,  2,..,  m) 

such  that  if 

m 


ij  -  t£  Cx.k  -  x.k)'] 


k=l 

then  d  .  <  d.  ..  whenever  p.  .  <  p,  ,  .    The  concept  of  the 
ij     kl  .  "ij    ^kl  r 

distant  function  p..  was  important  to  the  scaling  analysis 
and  deserves  amplification.   When  reading  as  computer 


Lingoes,  J.  C.,  New  Computer  Developments  in  Pattern 
Analysis  and  Nonmetric  Techniques,  paper  presented  at 
the  1964  IBM  Symposium  of  Statistics,  Paris:  Gauthier- 
Villas,  24  November  1966. 
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input  the  correlation  matrix  R,  the  distance  function  p.. 

refers  to  the  rank  of  correlation  coefficient  r. .  in  an 

ordered  column  vector  V.   For  example,  the  highest  r.. 

value  was  assigned  p, .  =  1,  the  second  highest  r, .  was 

ij  ij 

assigned  p.  .  =  2  etc.   The  meaning  of  'd.  .  <  d,  .,  whenever 
°  ij  ij     kl 

p.  .  <  p.  n  '  is  then  obvious, 
ij     kl 

The  practice  of  utilizing  only  the  ranks  of  the  co- 
efficients and  not  the  actual  values  when  determining 
optimal  point  orientation  on  the  multidimensional  plot  is 
the  nonmetric  method.   Nonmetric  rather  than  metric  con- 
straints permit  bypassing  strong  linear  and  distributional 
assumptions  required  of  metric  methods.   Defining  a  true 
metric  on  psychological  data,  as  was  the  case  in  this 
problem,  was  impossible.   To  enable  the  plotting  of  a 
meaningful  display  the  number  of  Euclidean  dimensions  is 
limited  to  m  =  2.   A  true  metric  constraint  denies  that 
possibility  due  to  the  strict  linear  distance  measure- 
ments between  points.   A  nonmetric  aonotonic  distance 
function  allows  for  the  reduction  to  a  two  dimensional 
display  while  still  maintaining  a  meaningful  plot. 
Several  goodness-of -f it  coefficients  used  in  this  study 
confirm  the  nonmetric  assumption  as  acceptable. 

A.   SMALLEST  SPACE  ANALYSIS  (SSA-I)  ALGORITHM 

The  algorithm  for  determing  the  optimal  orientation 
of  points  representing  the  n  objects  proceeds  as  follows. 
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STEP  1:   Read  in  the  data  matrix  R  of  inter-group 
correlation  values  or  the  data  matrix  T  of  inter-project 
dissimilarity  values. 

STEP  2:   Sort  the  data  values  from  large  to  small  if 
similarities  and  small  to  large  if  dissimilarities. 
Assign  ordered  ranks  of  1  to  n(n-l)/2  to  the  ordered 
vector.   Indicate  the  rank  of  element  r..  by  p... 

STEP  3:   Construct  a  matrix  C  by 


ij 


1  -  p±j/[n(n-l)/2] 


n 


n  -  V   c  •  • 


i=j 


Solve  for  the  Eigenvectors  and  Eigenvalues  of  C.   The 

set  of  Eigenvectors  x.   (i  =  1,  2,...,n;  a  =  1,  2,...,m) 

l  a 

is  the  initial  orientation  of  the  n  points  in  the  m 
dimensional  Euclidean  space. 

STEP  4:   Calculate  the  inter-point  distances  over 
the  m  dimensional  space. 

m 


k=l 


ij    VT;  '  "ik     J-K' 


Order  the  distances  d. .  in  a  row  vector  with  n(n-l)/2 
elements . 

STEP  5:   Redefine  the  p's  as  the  d's  permuted  to 
maintain  the  rank  order  of  the  original  distance  function 

and  designate  them  d*. 
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STEP  6:   Calculate  a  measure  of  effectiveness,  the 
phi  coefficient. 

n-1  n 

<{>  =   2  S     (d*.  -  d.  .)2 

1=1  j=l+l    ^     ^ 

STEP  7:   Calculate  a  new  C  matrix  on  the  basis  of 

the    d's    and    the    d*'s. 


ij 


1  +  S    <4/dik 

k=l 

,  -til 


i  =  J 


1*3 


d  .  . 

ij 

ft 

dii 

where  d.  .  =0,  d.  .  =0,  and  -: =  0  by  convection. 

ii         11  d  ,  . 

li 

STEP  8:   Calculate  a  new  set  of  point  coordinates 


X^J1  =  1/n  E     X*   c. . 

ik         ,  ,     lk   ii 
J=l 


k=l , 2 ,  .  . .  ,m 


STEP  9:   Calculate  again  the  inter-point  distances 
as  per  step  4  and  repeat  steps  5,  6,  7  and  8.   Continue 
until  the  normalized  phi  coefficient  indicates  further 
adjustment  of  points  will  not  advance  the  state  of  the 
inter-point  distance  and  original  distance  function 
relationship  toward  actual  monot onici ty . 

B.   GOODNESS-OF-FIT  MEASUREMENTS 

1 .   Gut tman-Lingoes  Coefficient  of  Alienation 

Consider  a  scattergram  of  n(n-l)/2  points  on 

ft 
coordinates   d. .  vs   d. .  .   Maximizing  a  statistic  U 
ij        13  6 
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is  defined  as  minimizing  the  alienation  of  plotted  points 

from  a  regression  line  through  the  origin.   The  coeffi- 

1/2 
cient  of  alienation  is  K  =  (1  -  y)    .   The  statistic  y 

* 
is  a  rank-order  correlation  coefficient  between  p..  =  d_,. 

* 

and  d.  .  •  y  equals  1  when  d.  .  =  a  d  .'.  for  some  constant 
of  proportionality  a  and  it  follows  that  the  coefficient 
of  alienation  K  would  equal  zero. 
2 .   Kruskal's  Stress 

Kruskal's  stress  is  derived  from  a  two-dimension- 

* 
al  plot  of  d. .  vs  p. .  and  is  a  statistic  quantifying  how 

well  the  given  configuration  of  points  represents  the 

data.   A  monotonic  increasing  piece-wise  linear  line  is 

drawn  connecting  thi  points.   The  points  d. .  are  also 

* 
plotted  on  the  graph.   The  distance  (d . .  -  d. .)  is  defined 

ij     ij 

as  the  horizontal  difference  between  the  actual  distance 
d.  .  and  the  desired  distance  d.  .  .   Kruskal  s  stress  is 
defined  as , 


E  cd 


S  = 


ij 


d*.)2 
3J 


S  <V 


Application  of  the  statistic  has  shown  that  for  any  value 
of  S  less  than  .20,  the  plotted  configuration  of  points 
is  a  good  representation  of  the  original  data.   Zero 
stress  indicates  complete  compliance  with  the  monotonic 
constraint . 
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V.   CALCULATION  RESULTS 

To  accomplish  the  objectives  of  this  study  of  compar- 
ing the  judge  groups  under  each  of  the  ranking  criteria, 
and  of  comparing  the  projects  being  ranked,  the  Smallest 
Space  Analysis  -  I  was  employed  several  times. 

A.   JUDGE  GROUP  COMPARISON 

1 .   Comparison  of  Judge  Groups  under  Criterion  A 

Columns  1  through  10  of  matrix  N  in  which  n. .  rep- 
resented  the  ranking  of  project  i  by  judge  group  j  under 
Criterion  A  (Military  Significance)  and  column  10  repre- 
senting fiscal  year  funding,  were  taken  for  the  first 
calculation.   The  correlation  matrix  R  was  calculated  for 
the  measure  of  similarity  between  judge  groups.   r.  .(i  =  l, 
2 , . . . , 10 ;  j-1,  2, ...,10)  was  then  used  as  input  to  SSA-I. 

Table  I  shows  the  resulting  point  positions  rep- 
resenting the  optimal  orientation  of  each  judge  group 
with  respect  to  the  other  nine.   The  values  for  Guttman- 
Lingoes'  coefficient  of  alienation  of  .290  and  Kruskal ' s 
stress  of  .243  were  both  marginal  values  and  indicated 
that  an  actual  monotonic  relationship  between  similari- 
ties and  inter-point  distances  was  not  attainable.   The 
plot  of  points,  Figure  1,  did  not  indicate  any  clustering 
or  significant  similarity  among  judge  group  rankings. 
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TABLE  I 
PLOT  OF  JUDGE  GROUPS 

RANKING  UNDER  THE  MILITARY  SIGNIFICANCE  CRITERION 

Judge  Group  Plot  Coordinates 

1  6.268     59.664 

2  -50-775    -69.316 

3  6.069    -16.553 

4  -100.000    -37.483 

5  -80.282     50.179 

6  100.000    -14.333 

7  19.227   -100.000 

8  36.415     32.546 

9  -26.288     20.693 
10  -59.613     -7.008 

Gut tman-Lingoes '  Coefficient  of  Alienation  =  0.29037 
Kruskal's  Stress  =  0.24326 
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Figure  1.   Plot  of  Judge  Groups  Ranking  under  the 
Military  Significance  Criterion. 
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2 .  Comparison  of  Judge  Groups  under  Criterion  B 
Columns  10  through  19  of  matrix  ft  with  n.  .  repre- 

senting  the  priority  rank  of  project  i  with  respect  to 
fiscal  year  funding  and  judge  groups  1  through  9  respec- 
tively, under  Criterion  B  were  used  for  the  second  run. 
The  correlation  values  r..  ( i=10 , 11 , . . . , 19 ;  j =10 , 11 , . . . 19) 
from  matrix  R  of  Appendix  D  was  the  input  to  SSA-I. 

Table  II  shows  the  resulting  point  positions  on 
the  two-dimensional  display.   The  values  for  Guttman- 
Lingoes'  coefficient  of  alienation  of  .257  and  for  Krus- 
kal ' s  stress  of  .214  were  again  marginal.   The  plot  of 
points  in  Figure  2,  by  being  very  scattered,  indicated 
that  none  of  the  judge  groups  agreed  to  a  significant 
degree  on  their  project  rankings. 

3 .  Comparison  of  Judge  Groups  under  Both  Criteria 
Neither  the  plot  of  judge  groups  under  Criterion 

A  nor  the  plot  of  judge  groups  under  Criterion  B  indica- 
ted a  strong  similarity  in  group  decisions.   By  plotting 
all  19  "groups"  together  it  would  be  possible  to  see  if 
any  groups  submitted  very  similar  rankings  under  both 
criteria.   Correlation  coefficients  r..  ( i  =  l ,  2  ,  .  .  .  ,  19  ; 

j=l,2 19)  from  matrix  R  of  Appendix  D  were  input  to 

SSA-I. 

Table  III  lists  the  plot  positions.   The  Guttman- 
Lingoes.' coef  f  icient  of  alienation  was  .190  and  the  Krus- 
kal  stress  was  .167.   The  values  indicated  an  excellent 
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TABLE  II 
PLOT  OF  JUDGE  GROUPS 

RANKING  UNDER  THE  INVESTMENT  RISK  CRITERION 

Judge  Group  Plot  Coordinates 

1  89.897  38.945 

2  33.497  -94.012 

3  -100.000  -45.473 

4  -83.266  57.879 

5  33.039  69.747 

6  100.000  -53.064 

7  . ,            16.167  -16.339 

8  -23.906  -100.000 

9  -1.639  23.583 

10                    -17.311  -  112.864 

Gut tman-Lingoes ' 

Coefficient  of  Alienation  =  0.25766 

Kruskal's  Stress  =  0.21421 
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Figure  2.   Plot  of  Judge  Groups  Ranking  under  the 
Investment  Risk  Criterion. 
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Judge  Group 

1 

2 

3 

4 

5 

6 

7 

8 

9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 


TABLE  III 
PLOT  OF  JUDGE  GROUPS 
RANKING  UNDER  BOTH  CRITERIA 

Plot  Coordinates 


-18.158 

-6.882 

51.030 

-90.134 

-21.517 

-8.888 

-64.799 

-96.362 

-100.000 

18.474 

74.592 

81.520 

86.038 

59.298 

66.162 

72.194 

-9.576 

62.696 

100.000 


24.269 
10.933 
-100 .000 
-24.549 
-80.273 
-40.472 
-59.218 
-48.063 
-94.551 
-83.546 
-7  .040 
-25.029 
-42.891 
-25.100 
-65.892 
-28.508 
-16.639 
-71.514 
-81.553 


Gut tman-Lingoes ' 

Coefficient  of  Alienation  =  0.19045 

Kruskal's  Stress  =  0.16754 
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Figure  3.   Plot  of  Judge  Groups  Ranking  under  Both 
Criteria . 
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plot  in  terms  of  variation  from  the  regression  line  for 
the  G-L  coefficient  as  detailed  in  paragraph  IV  C  2.   The 
Kruskal  stress  also  indicated  excellent  correlation  of 

plotted  values  with  the  monotonically  increasing  line  on 

A 
p. .  vs  d. .  as  explained  in  paragraph  IV  C  2. 
ij      ij 

Figure  3  was  not  applicable  to  clustering  consid- 
eration because  it  represented  the  judge  groups  as  nine- 
teen separate  entities,  i.e.,  each  judge  group  was 
represented  twice.   However,  one  thing  was  clearly  indi- 
cated.  Each  judge  group  submitted  significantly  differ- 
ent rankings  for  each  of  the  two  criteria.   With  the 
exception  of  group  3  under  Criterion  A  (point  3)  and 
group  7  under  Criterion  B  (point  17)  ,  there  was  no  over- 
lapping of  the  judge  group  rankings.   The  results  indica- 
ted that  the  judges  actually  did  consider  the  criteria 
of  Military  Significance  and  Investment  Risk  when  ranking 
their  projects.   That  is  an  important  argument  for  data 
validity . 

B.   PROJECT  COMPARISONS 

The  second  objective  of  this  study  was  to  compare  the 
projects  and  examine  possible  groups  of  projects  that  the 
judges  considered  of  equal  importance  or  that  judge  group 
consensus  rated  high  or  low. 

1 .   Comparison  of  Projects  Ranked  under  Criterion  A 

The  eighteen  projects  as  ranked  under  Criterion  A 
plus  the  fiscal  year  funds  ranking  were  the  first  of  the 
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project  groups  considered.   The  entries  of  matrix  T   in 

Appendix  E  are  values  of  the  interpoint  dissimilarity 

measure  t    for  the  Military  Significance  Criterion. 

The  input  to  SSA-I  was  the  matrix  T , . 

A 

Table  II  shows  the  point  position  representing 
the  projects  on  the  two-dimensional  plot.   The  value  of 
Gut tman-Lingoes '  coefficient  of  alienation  was  .067  and 
Kruskal/s  stress  was  .053.   Both  values  indicate  that  the 
resulting  plot  was  an  excellent  reflection  of  the  origi- 
nal data  as  determined  by  the  distance  function  p... 
Figure  4  showed  definite  clusters  of  projects.   The  rule 
chosen  for  defining  a  cluster  of  c  points  was  that  each 
point  be  within  20.0  units  of  the  other  c-1  points  of  the 
cluster.   (The  longest  possible  inter-point  distance  on 
the  plot  was  282.0  units.)   Using  that  rule,  2  major 
clusters  of  3  or  more  points  were  established.   One 
cluster  was  composed  of  projects  5,  8  and  18,  and  the 
other  cluster  was  composed  of  projects  3,  10,  12,  14 
and  15.   Minor  clusters  were  established  between  pro- 
jects 7  and  9,  and  between  projects  4  and  16.   The 
implication  drawn  from  the  clusters  was  that  the  pro- 
jects in  each  cluster  were  considered  of  equal  priority 
by  the  judge  groups  when  ranking  under  the  Military 
Significance  Criterion.   The  SSA-I  plot  is  more  meaning- 
ful than  a  standard  correlation  statistic  because  each 
point  was  plotted  with  respect  to  the  other  n-1  points. 
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TABLE  IV 
PLOT  OF  PROJECTS 
WHEN  RANKED  UNDER  THE  MILITARY  SIGNIFICANCE  CRITERION 

Plot  Coordinates 

88.636  -46.802 

-80.882  22.898 

5.479  -82.690 

50.223  -48.019 

-34.760  -52.799 

-16.022  -75.868 

38.913  -67.702 

-41.815  -56.905 

30.572  -66.946 

16.792  -66.719 

-81.456  -100.000 

3.653  -77.032 

100.000  -26.187 

11.188  -74.394 

16.804  -87.316 

55.806  -37.771 

-100.000  -51.790 

-51.634  -49.414 

Gut tman-Lingoes '  Coefficient 
of  Alienation  =  0.06728 

Kruskal's  Stress  =  0.05308 
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Figure  4.   Plot  of  Projects  when  Ranked  under  the 
Military  Significance  Criterion. 
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A  correlation  statistic  reflects  the  collation  between 
two  points  regardless  of  each  point's  relationship  to 
the  remaining  points. 

2  .   Comparison  of  Projects  under  Criterion  B 

The  entries  of  matrix  T,.  in  Appendix  E  were  used 

o 

as  the  SSA-I  input  for  comparing  projects  as  they  were 
ranked  by  the  judges  under  the  Investment  Risk  Criterion. 

Table  V  shows  the  point  positions,  the  Guttman- 
Lingoes'  coefficient  of  alienation  as  .030  and  the  Kruskal 
stress  as  .021.   Again  the  values  indicated  an  excellent 
fit  with  respect  to  the  plotting  constraints.   Figure  5 
showed  the  project  point  orientation.   Using  the  rule 
that  inter-point  distances  be  less  than  20.0  units  for 
each  cluster  point,  three  major  clusters  were  established. 
One  cluster  was  composed  of  projects  3,  15,  17  and  18; 
another  was  composed  of  projects  1,  13  and  14,  and  the 
third  of  projects  6,  7  and  10.   A  minor  cluster  between 
projects  12  and  16  was  also  in  evidence.   Therefore,  rank- 
ing on  the  basis  of  Investment  Risk  the  judges  felt  the 
clustered  projects  entailed  similar  risk. 

3.   Comparison  of  Projects  using  Both  Criteria 

In  actual  practice  both  Military  Significance  and 
Investment  Risk  are  considered  when  allocating  research 
resources.   For  that  reason  a  plot  of  the  projects  consid- 
ering both  criteria  was.  desired.   Matrix  TAB  of  Appendix  E 
was  used  as  the  input  for  SSA-I.   It  was  derived  by  con- 
sidering the  judges  as  nineteen  separate  groups-. 
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Table  VI  displays  the  point  positions.   The  Gutt- 
man-Lingoes '  coefficient  of  alienation  of  .026  and  the 
Kruskal  stress  of  .018  both  indicated  an  excellent  plot 
in  terms  of  meeting  the  plotting  algorithm  constraint 
criteria.   Figure  6  showed  several  clusters,  the  major 
ones  being  projects  5,  8,  17  and  18,  and  projects  4,  6 
and  7.   Minor  clusters  consisted  of  points  9  and  11,  points 
3  and  15,  points  12  and  14,  and  points  1  and  16.   Consider- 
ing the  projects  under  both  ranking  criteria,  those 
clusters  represented  the  projects  which  the  judge  groups 
felt  should  receive  equal  emphasis  for  research. 
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TABLE   V 
PLOT  OF  PROJECTS 

WHEN  RANKED  UNDER  THE  INVESTMENT  RISK  CRITERION 

Plot  Coordinates 

81.826  -31.333 

-85.361  15.915 

-54.677  -66.792 

-68.344  -52.507 

-100.000  -29.425 

-18.818  -74.158 

-30.314  -64.395 

-67.295  -76.211 

-82.586  -4.289 

-20.116  -87.728 

26.285  -100.000 

71.931  -43.641 

100.000  -23.824 

94.434  -27.973 

-48.304  -61.729 

68.200  -54.758 

-45.503  -70,721 

-43.436  -64.363 

Guttman-Lingoes ' 
Coefficient  of  Alienation  =  0.03073 

Kruskal's  Stress  =  0.02190 
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Figure  5.   Plot  of  Projects  when  Ranked  under  the 
Investment  Risk  Criterion. 
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TABLE  VI 
PLOT  OF  PROJECTS 
WHEN  RANKED  UNDER  BOTH  CRITERIA 

Plot  Coordinates 

84.667  -38.574 

-100.000  46.714 

-55.133  -88.801 

-30.757  -83.865 

-89.757  -35.684 

-37.145  -96.417 

-21.369  -88.825 

-96.979  -66.706 

-58.968  -63.651 

-13.578  -100.000 

-66.832  -64.756 

29.745  -75.383 

100.000  -9.750 

34.864  -67.964 

-50.375  -90.462 

77.295  -52.623 

-96.708  -47.245 

-85.638  -59.449 

Guttman-Lingoes ' 
Coefficient  of  Alienation  =  0.02633 

Kruskal's  Stress  =  0.01814 
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Figure  6.   Plot  of  Projects  when  Ranked  under  Both 
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VI  .   CONCLUSIONS 

Results  of  the  SSA-I  plots  of  judge  groups  indicated 
that  no  two  judge  groups  were  in  strong  agreement  when 
considering  their  rankings  as  a  whole.   Groups  often 
agreed  on  which  projects  should  be  ranked  high  under  one 
criterion  or  the  other  but  disagreed  on  which  projects 
should  be  ranked  low.   The  reverse,  agreeing  on  the  lower 
ranked  projects  but  disagreeing  on  the  higher  ranks,  was 
also  in  evidence. 

Different  conclusions  can  be  drawn  from  the  project 
comparisons.   The  three  plots  using  Criterion  A,  Crite- 
rion B,  and  both  Criteria  A  and  B,  all  displayed  cluster- 
ing tendencies.   To  display  the  results  in  an  easy-to-read 
fashion  the  following  was  done.   For  each  cluster,  the 
mean  rank  of  the  projects  in  that  cluster  was  calculated 
and  designated  the  mean  cluster  rank.   A  ladder  was  con- 
structed of  ordered  mean  ranks  for  both  clusters  and  non- 
clustered  projects.   The  result  is  how  the  judge  groups 
felt  the  projects  should  be  ranked  considering  the  three 
criteria  above.   Figure  7  displays  the  resulting  project 
priority  ladders. 

Investigation  of  the  variation  in  project  rankings 
(Appendix  E)  shows  which  project  was  ranked  most  consis- 
tent by  the  judge  groups.   That  project  with  the  lowest 
standard  deviation  of  ranking  was  project  17  with  a  mean 
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rank  of  4.70  and  a  standard  deviation  of  1.68  for  Crite- 
rion A,  project  13  with  a  mean  rank  of  15.90  and  a 
standard  deviation  of  1.87  for  Criterion  B,  and  project 
13  with  a  mean  rank  of  15.74  and  a  standard  deviation 
of  2.27  for  Criteria  A  and  B  together. 
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Manpower  Requirements 
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Personnel  Performance 
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Occupational  Research  and 
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Organizational  Effectiveness 
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Man-Machine  System  R&D 
Applied  Systems  Development 

Manpower /Per sonnel  Cost  Systems 
R&D  Resources  Management 

Drug  Abuse 

Minority  Discrimination 
Measurement  and  Reduction 


Figure  7a.   Project  Priority  Ladder  as  a  Result  of 
Clustering  under  the  Military  Significance  Criterion 
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Recruitment /Procurement 
Personnel  Performance 

Personnel  Management  Systems 
Man-Machine  Systems  R&D 
Career  Structures 

Personnel  Retention 

Attitude/Motivation 

R&D  Resources  Management 

Drug  Abuse 
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Measurement  and  Reduction 
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Figure  7b.   Project  Priority  Ladder  as  a  Result  of 
Clustering  under  the  Investment  Risk  Criterion. 
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Figure  7c.   Project  Priority  Ladder  as  a  Result  of 
Clustering  under  both  Criteria;  Military  Significance 
and  Investment  Risk. 
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APPENDIX  A 
LIST  OF  JUDGE  GROUPS 

Group  Number      Group  Composition 

1  Manager  personnel,  Pers  A-3 

2  Manager  personnel,  Headquarters,  Dept 
of  the  Navy 

3  Manager  personnel,  Naval  Personnel 
Laboratory,  Washington,  D.  C. 

4  Professional  worker,  Naval  Personnel 
Laboratory,  San  Diego 

5  Professional  worker,  Naval  Personnel 
Laboratory,  Washington,  D.  C. 

6  Manager  personnel,  Naval  Personnel 
Laboratory,  San  Diego 

7  Members,  user  panel  (students  at  the 
Naval  Postgraduate  School) 

8  Members,  Laboratory  Advisory  Board 
for  Personnel  Laboratories 

9  Technical /Pro f ess ional  persons,  con- 
tractor activities 

10  Fiscal  year  1973  dollar  funding  for 

the  specified  project 
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APPENDIX  B 
LIST  OF  PROJECTS  STUDIED 

This  Appendix  lists  the  projects  studied  by  identi- 
fication number  and  short  ,title,  and  includes  a  paragraph 
describing  each.   This  information  was  available  to  each 
judge  when  he  compiled  his  ranking. 

Project  No.   Short  Title 

1  Drug  Abuse 

This  area  consists  of  research  and  develop- 
ment for  (a)  determining  causes,  attitudes  toward,  and 
extent  of  drug  abuse  in  the  Navy;  (b)  exploration  of  means 
to  reduce,  prevent,  and  control  drug  abuse;  and  (c)  means 
to  provide  effective  education,  counseling,  and  rehabili- 
tation programs  for  Naval  personnel  and  dependents. 

2  Education  and  Training 

The  two  mutually  supporting  purposes  of 
Education  and  Training  research  and  development  are  (a) 
improving  Navy  training  and  (b)  development  of  principles 
and  technology  of  training.   Navy  requirements  call  for 
more  accurate  definition  of  training  requirements  and 
for  training  methods  directed  toward  meeting  future  job 
requirements.   Research  and  development  in  this  area  in- 
volves work  on  the  broad  spectrum  of  problems  related 
deriving  and  specifying  appropriate  training  content  and 
developing  and  evaluating  ways  to  improve  acquisition 
of  training  objectives.   Research  in  this  area  also 
includes  specific  problems  of  learning,  perception, 
job  aids,  programming,  displays,  simulation,  and  manage- 
ment as  they  relate  to  the  improvement  of  training. 

3  Occupational  Research  and  Development 

Development,  test  and  evaluation  of  new 
and  improved  techniques  for  describing,  evaluating, 
structuring,  and  re-engineering  Navy  occupations  for 
optimal  matching  of  available  manpower  characteristics 
to  changing  job  requirements.   Involves  new  methods  for 


46 


determining  skills  required  for  effective  performance  of 
occupations  and  the  storage  and  retrieval  of  occupational 
data  pertaining  to  work  requirements  of  Navy  personnel. 

4  Manpower /Per sonnel  Cost  Systems 


and  improv 
tion ,  s tor 
data;  (b) 
mixes  of  m 
and  person 
permit  man 
and  effect 
The  princi 
costs  from 
be  used  to 
total  Navy 


Development,  test  and  eva 

ed  (a)  techniques  for  the  acquis 

age,  and  retrieval  of  manpower/p 

techniques  for  determining  the  c 

en  and  equipment,  personnel  mana 

nel  utilization  factors;  and  (c) 

power  requirements  planners  to  t 

iveness  in  order  to  evaluate  ava 

pal  objective  is  to  furnish  tota 

procurement  through  post-separa 

formulate  cost-related  decision 

manpower  personnel  systems. 


luation  of  new 
ition,  computa- 
ersonnel  cost 
ost  of  various 
gement  policies, 

measures  which 
rade  off  cost 
ilable  options. 
1  government 
tion  which  can 
s  concerning  the 


5  Manpower  Requirements 

Development,  test  and  eval 
techniques  for  defining  and  displaying  quan 
qualitative  manpower  requirements  for  all  f 
formed  ashore  and  afloat  including  upward  a 
requirements  to  the  total  Navy  establishmen 
niques  must  be  responsive  to  the  changing  o 
requirements,  system  design  innovations,  an 
human  resources.  Includes  the  development  o 
productivity  models  and  the  use  of  Simula ti 
to  test  alternative  manning  and  manpower  ma 
cepts  in  terms  of  cost  and  total  system  eff 
The  principal  objective  is  to  develop  metho 
mining  manpower  requirements  which  are  capa 
ing  quickly  to  changes  in  resources  and  mil 
and  requirements. 


uation  of 

t  itat  ive  and 
unctions  per- 
ggregation  of 
t  .   The  te  ch- 
perational 
d  available 
f  manpower/ 
on  techniques 
nagement  con- 
e  ct  ivenes  s . 
ds  for  deter- 
ble  of  respond- 
itary  objectives 


6  Personnel  Management  Systems 

Development,  test  and  evaluation  of  tools 
to  improve  the  quality  of  personnel  management  by  increas- 
ing the  accuracy  of  force  projections,   by  responding  more 
rapidly  to  changing  policies  and  operational  conditions, 
and  by  projecting  the  effects  of  current  and  proposed  per- 
sonnel policies  to  determine  the  most  effective  personnel 
management  strategies.   These  tools  are  usually  in  the  form 
of  computerized  models  of  the  personnel  system  which  can  be 
used  to  reflect  the  relationship  between  such  variables  as 
recruit  input,  advancement,  and  attrition  and  to  project  the 
effect  of  these  variables  upon  the  size  and  skill  composi- 
tion of  the  personnel  force. 
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Man-Machine  R&D 


implementat 
development 
rent  organi 
include  (a) 
and  extrapo 
formance,  e 
conditions ; 
standards , 
gies  in  sys 
(c)  techniq 
anticipated 

8 


This  area  is  primarily  conce 
ion  of  human  operator  consideratio 
,  operations,  and  maintenance  of  n 
zational,  weapons  and  support  syst 

an  understanding  of,  and  techniqu 
late,  meaningful  relationships  amo 
quipment  characteristics,  and  envi 

(b)  development  and  application  o 
specifications,  and  tradeoff  analy 
terns  development,  planning,  and  pr 
ues  for  application  of  human  capab 

demand  of  future  weapon  systems. 

Selection/Classification 


rned  with  the 
ns  in  the 
ew  and  cur- 
ems  .   Goals 
es  to  quantify 
ng  human  per- 
r onmental 
f  man-machine 
sis  methodolo- 
ocurement ;  and 
ilities  to 


The  program  area  involves  development, 
test  and  evaluation  of  new  and  improved  methods  for  (a) 
more  accurate  and  effective  selection  and  classification 
of  individuals;  (b)  determining  their  training  requirements; 
and  (c)  matching  the  man  to  the  job.   Selection  usually 
refers  to  the  process  of  determining  which  individual  or 
individuals  should  be  accepted  or  rejected  with  regard  to 
a  specific  program.   Classification  refers  to  determining 
and  categorizing  the  appropriate  skill  qualifications  of 
an  individual  that  each  school  or  job  quota  is  filled 
with  particular  individuals  most  likely  to  perform  success- 
fully. 

9  Applied  Systems  Development 

Identification  of  personnel  and  training 
requirements,  establishment  of  initial  operator  performance 
standards  and  maintenance  skill  requirements,  assistance 
in  man-machine  tradeoff  studies,  and  conducting  studies  to 
enhance  man-machine  compatibility  for  candidate  and  proto- 
type systems  under  development.   The  effort  frequently 
involves  development  of  training  objectives,  review  of 
course  and  lesson  plans,  and  the  evaluation  of  operator 
and  maintenance  personnel  training. 
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Career  Structure 


Development,  test  and  evaluation  of  new 
and  improved  techniques  for  (a)  arriving  at  optimal  career 
progression  patterns;  (b)  methods  for  utilizing  occupational, 
manpower /personnel  cost,  and  other  manpower  factors  data 
in  determining  optimal  combination  of  grade  and  skill 
profiles  required  to  satisfy  Navy  operational  requirements; 
and  (c)  techniques  for  providing  balance  between  needs  of 


48 


the  service  based  on  technological  projections,  and  needs 
of  the  individual  related  to  opportunities  for  increased 

„ !  V  J    1    J  J „J-.._~ „  „  ,,  »- „  ±.  J    _  „      J    j_  ■ 


11  Personnel  Retention 

This  is  a  multi-faceted  program  exploring 
and  developing  methods  and  techniques  for  inducing  quali- 
fied and  needed  personnel  to  remain  on  active  duty,  and 
programs  which  meet  fluctuating  needs  of  the  naval  service 
Included  are  (a)  investigation  in  psychometric  technology 
to  predict  retention  for  use  in  selection  and  assignment; 
(b)  development  of  more  fair  and  effective  evaluation  of 
performance  through  computer-aided  analysis  and  display  of 
performance  evaluation;  and  (c)  development  of  improved 
techniques  for  selection  and  training  of  personnel  in 
ratings  which  administer  personal  services  to  other  naval 
personnel . 

12  Attitude/Motivation 

Develop  sociometric  and  psychometric  in- 
struments and  conduct  controlled  experiments  for  purposes 
of  studying  man's  values,  needs,  aspirations,  prejudices 
and  expectations  to  achieve  a  better  understanding  of  how 
they  affect  his  thinking,  feelings,  and  reactions  to 
people,  groups,  social  issues  or  events  in  his  environment 
The  objective  in  achieving  such  an  understanding  is  better 
prediction  and  control  of  an  individual's  behavior,  moti- 
vation and  performance. 

13  Minority  Discrimination  Measurement  and 
Reduction 

This  area  is  comprised  of  R  &  D  support 
for  BuPers  and  Navy  objectives  to  intensify  and  expand 
efforts  to  eliminate  racial,  sex,  and  cultural  discrimina- 
tion and  bias  in  the  Navy. 

14  Organization  Effectiveness 

Development  and  evaluation  of  techniques 
for  measuring  overall  military  effectiveness  of  organiza- 
tional units  under  various  contingencies.   This  involves 
the  development  of  empirically  based  criteria  and  the 
relative  importance  of  criteria  which  should  be  applied 
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in  making  judgments  of  organizational  effectiveness.   It 
includes  studies  of  individual  and  organizational  behavior 
and  evaluative  research  into  managerial  practices  and  or- 
ganizational characteristics.   The  principal  objective  is 
the  enhancement  of  organizational  effectiveness  in  rela- 
tion to  organizational  goals  and  total  system  effectiveness. 
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Distribution/Assignment 


This  area  enc 
development  in  the  managerial 
distribution,  assignment,  and 
the  Bureau  of  Naval  Personnel, 
with  the  development  and  evalu 
techniques  and  procedures  base 
technology  for  (a)  the  equitab 
qualitative  allocation  ("distr 
and  inventory  among  the  ships, 
shore  stations  of  the  Navy;  (b 
("assignment")  of  specific  ind 
billets  in  terms  of  prescribed 
planned,  periodic  reassignment 
personnel  to  the  various  categ 
overseas  duty.  Objectives  are 
of  personnel  and  billets,  impr 
systems,  and  optimal  matching 
and  resources  over  time. 


ompasses  research  and 
functions  of  personnel 
rotation  as  carried  out  by 
The  program  is  concerned 
ation  of  new  and  improved 
d  on  modern  computer 
le ,  quantitative  and 
ibution")  of  the  personnel 

aircraft  squadrons,  and 
)  the  optimal  match 
ividuals  to  particular 

criteria;  and  (c)  the 

("rotation")  of  career 
ories  of  sea,  shore  and 

the  optimum  matching 
oved  manpower  utilization 
of  manpower  requirements 
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R&D  Resources  Management 


This  area  includes  three  sub 
with  efficient  use  of  R  &  D  resources.  One  d 
the  selection  and  evaluation  of  scientific/re 
sonnel  employed  by  the  laboratories.  Researc 
area  is  aimed  at  the  development  of  performan 
criteria  in  the  area  of  quantitative  and  qual 
work  output  and  s cient if ic/ technical  contribu 
Another  subarea  concerns  the  identification  o 
problem  areas,  assignment  of  priorities,  and 
tasks  to  be  performed  to  assure  maximum  retur 
D  dollar".  A  third  subarea  is  the  continuing 
implementation  of  research  recommendations  by 
a  basis  for  future  research  planning  and  to  f 
implementation  of  significant  R&D  products. 


areas  dealing 
eals  with 
search  per- 
il in  this 
ce  evaluation 
i  tat ive 
t  ions . 

f  significant 
s  election  of 
n  for  the"R  & 
study  of  the 
consumers  as 
acilit ate  the 


17 


Recruitment /Procurement 


Development,  test  and  evaluation  of 
methods  and  techniques  for  (a)  attracting  and  obtaining 
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required  skill  composition,  racial  mix,  and  educational 
levels  and  for  (b)  improving  selection  policies  and 
programs  relating  to  manpower  requirements,  implications 
of  an  all  volunteer  force,  manpower  ceilings  reductions, 
and  changes  in  strategic  guidance.   The  principal 
objective  is  to  obtain  personnel  with  the  appropriate 
innate  and  acquired  characteristics  to  meet  all  of  the 
performance  requirements  and  changing  requirements  of 
future  naval  forces. 
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Personnel  Performance 


Development  of  on-the-job  performance 
standards  and  criteria  and  improved  proficiency  measure- 
ment techniques  to  assure  that  selection,  assignment, 
and  promotion  factors  are  job-relevant.   The  goal  is 
maximum  readiness  of  individuals  and  units. 
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APPENDIX  C 


MATRIX  M 
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APPENDIX  D 
JUDGE  GROUP  CORRELATION  MATRIX  R 
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1.00 
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1.00 
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.29 

.30 

1.00 

4 

.40 

.38 

.14 

1.00 

5 

.53 

.55 

.38 

.47 

1.00 
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.60 

.69 

.41 

.65 

.69 

1.00 

7 

.45 

.47 

.36 

.65 

.59 
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.24 

.36 
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.69 
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.41 
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.25 
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.53 
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.49 

.04 
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.69 

.02 

.26 

.56 
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.01 
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13 

.11 
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.68 
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.13 

-.03 

-.13 

14 

.37 
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.26 

.65 

.19 

-.03 

-.01 

15 

.30 

.46 

.64 

.00 

.59 

.54 
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-.  11 
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.35 

.31 
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.63 
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-.10 

-.03 
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.47 

.46 

.27 
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.53 

.23 
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.36 

.65 

.04 
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-.02 

-.23 
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10   ■  11     12     13     14     15     16     17     18 
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.36 
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APPENDIX  E 
DATA  ON  INDIVIDUAL  PROJECT  RANKINGS  FOR  SPECIFIED  CRITERION 

Criterion  A      Criterion  B      Criteria  A&B 


roj .No . 

Mean 

STD 

Mean 

STD 

Mean 

STD 

1 

14.50 

2.94 

15.40 

2.62 

14.89 

2.88 

2 

3.00 

2.28 

3.00 

2.79 

3.11 

2.57 

3 

9.70 

3.74 

7.30 

3.61 

8.58 

3.95 

4 

13.00 

4.60 

6.50 

3.11 

9.53 

5.13 

5 

6.90 

5.13 

5.00 

1.90 

5.95 

4.08 

6 

8.60 

3.61 

9.30 

4.29 

9.32 

3.74 

7 

11.70 

3.93 

8.50 

4.54 

9.95 

4.61 

8 

7.10 

3.14 

7  .10 

2.17 

6.89 

2.61 

9 

11.30 

4.  78 

3.80 

3.16 

7.74 

5.60 

10 

10.50 

4.82 

9.40 

3.17 

10.32 

3.89 

11 

3.80 

3.54 

11.70 

2.69 

7.53 

5.08 

12 

9.60 

4.10 

14.90 
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12.47 

4.50 

13 
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15.74 

2.27 
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4.41 
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7  .50 

4  .  1 5 

8.  74 

4.00 
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13.80 

4.77 

14.40 

3.04 

14  .42 

3.10 

17 

4.70 

1.68 

7.80 

3.54 

6.32 

3.25 

18 

6.10 

3.75 

7  .70 

4.15 

6.79 

4.11 
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3.  ABSTRACT 


The  problem  is  defined  as  examining  a  list  of  the 
research  projects  presently  sponsored  by  the  Department 
of  the  Navy  under  GOR-43  and  analyze  the  ordered  project 
listings  as  ranked  by  panels  composed  of  persons  with 
interest  in  those  projects.  *,ino9    nf  193 

Data  consisted  of  project  priority  rankings  of  193 
iudges  composing  9  separate  panels,  each  judge  ranking 
he8   o   cs  twice  in  priority  order;  once  considering 
the  Practical  significance  of  the  project  and  once  con- 
e  possibility  of  a  successful  research  effort 

'rankings  of  judges  within  each  panel  were  scaled 
using  the  FORD  procedure  and  the  9  panel  ran  in 

*.  •   *-^a     f^t-  similarities  using  methods  o±  cluster 
investigated  tor  simiiai  j-  l  j-  =  o  <-.    b 

^^"esults  showed  that  no  2  judge  panels  submitted 
highly  correlated  rankings  of  the  18  projects,  but  that 
under  each  criterion  several  project  clusters  were  evi- 
dent! indicating  equal  research  emphasis  desired  by  the 
panels . 
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